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(54) DIGITAL PICTURE SIGNAL PROCESSING UNIT 

(57)Abstract: 

PURPOSE: To suppress the increase in a memory capacity storing data for 
classification as less as possible even when the number of reference picture elements 
is increased by classifying an input picture in a way of representing a local 
characteristic of the input picture in the case of processing such as conversion of an 
input picture signal into a picture signal with higher definition. 

CONSTITUTION: A scanning conversion circuit 2 outputs plural reference picture 
element data simultaneously in the vicinity of a noted picture element spacially 
timewise. A class prediction device 3 forms a prediction value with respect to a 
pi^ecletermined SD picture element by linear combination between a class prediction 
coefficient from a table 5 and a reference picture element. A class decision circuit 4 
detects a minimum value in errors between the prediction value calculated for all 
classes and a real value of predetermined SD picture element. The class 



corresponding to the minimum value is decided as a class of the noted picture 
element. Furthermore, a data prediction coefficient is stored in the table 5 for each 
class. The HD picture signal is calculated by linear combination between the data 
prediction coefficient and the plural SD picture element. 

* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect 
the original precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1]A digital image signal processor which needs to create a value of a noticed 
picture element characterized by comprising the following using two or more pixels of 
the above-mentioned noticed picture element which exist in the neighborhood 
spatially and/or in time. 

A table where two or more class prediction coefficients were beforehand stored for 
every class. 

It is contained in an input digital image signal, and spatially and/or in time to the 
above-mentioned noticed picture element by primary combination of a class 
prediction coefficient from two or more nearby reference pixels and above-mentioned 
tables. A class sorting means for determining a class of the above-mentioned noticed 
picture element by forming a predicted value of predetermined picture element data of 
the above-mentioned input digital image signal, and identifying the above-mentioned 
class prediction coefficient which produces a true value of the above-mentioned 
predetermined pixel, and the nearest above-mentioned predicted value. 
A means for generating a value for data prediction for creating a value of the 
above-mentioned noticed picture element for every class determined 
[ above-mentioned ]. 

[Claim 2]In a digital image signal processor which needs to create a value of a noticed 
picture element using two or more pixels of the above-mentioned noticed picture 



element which exist in the neighborhood spatially and/or in time. Are a learning device 
for determining two or more class prediction coefficients for every class, and it is 
contained in an input digital image signal, and spatially and/or in time to the 
above-mentioned noticed picture element by primary combination of two or more 
nearby reference pixels and a class prediction coefficient. A learning device which 
determined the above-mentioned class prediction coefficient with the least square 
method so that an error of the above-mentioned predicted value and a true value of 
the above-mentioned predetermined pixel might be made into the minimum when a 
predicted value of predetermined picture element data of the above-mentioned input 
digital image signal was formed. 

[Claim 3]A digital image signal processor characterized by comprising the following for 
changing the 1st picture signal with low resolution into the 2nd higher-resolution 
picture signal. 

A table where two or more class prediction coefficients were beforehand stored for 
every class. 

To the above-mentioned noticed picture element in a picture signal of the above 2nd, 
spatially and/or in time by primary combination of a class prediction coefficient from a 
reference pixel and the above-mentioned table in a picture signal of the above 1st of 
nearby plurality. A class sorting means for determining a class of the 
above-mentioned noticed picture element by forming a predicted value of 
predetermined picture element data of the 1st picture signal of the above, and 
identifying the above-mentioned class prediction coefficient which produces a true 
value of the above-mentioned predetermined pixel, and the nearest above-mentioned 
predicted value. 

A means for answering class determination by a means and the above-mentioned 
class sorting means for generating a value for data prediction for creating a value of 
the above-mentioned noticed picture element for every class determined 
[ above-mentioned ], and creating a value of the above-mentioned noticed picture 
element using data ** data of a determined class. 

[Claim 4]In claim 1 and the digital image signal processor according to claim 2 or 3, a 
means for generating a value for the above-mentioned data prediction, It is contained 
in the 1st picture signal and the above-mentioned noticed picture element spatially 
and/or in time by primary linearity combination of a value of two or more nearby pixels, 
and two or more coefficients. A digital image signal processor generating a coefficient 
which makes the minimum an error of a created value and a true value of the 
above-mentioned noticed picture element for every above-mentioned class when a 
value of the above-mentioned noticed picture element is created. 
[Claim 5]A digital image signal processor, wherein a means for generating a value for 
the above-mentioned data prediction generates a central value of the 



above-mentioned noticed picture element beforehand gained by study for every 
above-mentioned class in claim 1 and the digital image signal processor according to 
claim 2 or 3. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application]This invention relates to the digital image signal processor 
which needs to create the value of a noticed picture element using two or more pixels 
which exist in the neighborhood spatially and/or in time. 
[0002] 

[Description of the Prior Art]As one of the high efficiency coding of a digital image 
signal, there are some which decrease transmission data volume by thinning out a 
pixel by subsampling. The example is a multiplex Sub-Nyquist-sampling encoding 
method in MUSE. It is thinned out in a receiver and it is necessary to interpolate the 
pixel of non transmission in this system. The rise conversion which changes the video 
signal of a standard definition inputted into the video of a high definition is also 
proposed. In this case, it is necessary to create the pixel which run short from the 
signal of a standard definition. When expanding a picture electronically, interpolation of 
the value of the pixel running short is needed. It is necessary to create the point 
estimate of a noticed picture element from the value of a surrounding pixel not only in 
these things but in scene change detection and DPCM. 

[0003]As mentioned above, when creating the value of a noticed picture element, 
usually at the former, the interpolation filter of the fixed tap and the fixed factor was 
used. 
[0004] 

[Problem(s) to be Solved by the Invention] even if the processing which interpolates a 
non transmission pixel with an interpolation filter is effective to a certain kind of 
picture, on the whole, interpolation processing is effectively demonstrated about the 
picture of various kinds, such as a picture with a motion, and a still picture, — 
9|c9|o|e:ic9|c^)i»ie , As the result. the problem which the "jerkiness" etc. which is the 
unnaturalness of "dotage" and a motion generates was in the image restoration which 
comprises a transmission pixel and an interpolating picture element. 
[0005]As one method of solving this problem, the value of a noticed picture element is 
expressed with primary linearity combination of the pixel of the circumference of it. 
and a coefficient, and what determines the value of this coefficient with the least 
square method using the actual value of a noticed picture element is proposed so that 



a square with error may serve as the minimum. It cannot be said that it can form the 
interpolation value which fully reflected the local features of the picture containing a 
noticed picture element although this method is effective. 

[0006]Since the local features of a picture are reflected, performing class sorting 
based on the surrounding level distribution of a noticed picture element is proposed. 
As a generation method of this class, what uses the level of picture element data as it 
is can be considered. By the case where 4 pixels is used for class sorting, the class of 
\2 ®) =2 is needed, and this method has a problem to which the capacity of the 
memory which stores the data for class sorting becomes large too much, when picture 
element data is expressed by 8 bits. 

[0007]To JP,63-48088,A by an applicant's for this patent proposal. When interpolating 
an infanticide pixel, the average value of a surrounding reference pixel is calculated, 
each pixel is expressed by 1 bit according to the size relation of average value and the 
value of each pixel, and performing the class division according to the pattern of 
(number xof reference pixels 1 bit) is proposed. However, since this method is what 
binary-izes the value of each pixel, it becomes insufficient [ reflecting the local 
features of a picture correctly ]. When the data of a reference pixel is compressed for 
class sorting and a compression ratio is made high, there is same problem. 
[0008]Therefore. when one purpose of this invention performs class sorting. An error 
with the actual value which a noticed picture element has is able to create few values. 
Even when there are many reference pixels, the capacity of the memory which 
memorizes the data for class sorting is comparatively small, and it is in providing 
further the digital image signal processor which can perform class sorting correctly. 
[0009]Other purposes of this invention are to provide the digital image signal 
processor which can change a picture signal with low resolution into a high-resolution 
picture signal (rise conversion). 
[0010] 

[Means for Solving the Problem]In a digital image signal processor which needs for the 
invention according to claim 1 to create a value of a noticed picture element using two 
or more pixels of a noticed picture element which exist in the neighborhood spatially 
and/or in time, Two or more class prediction coefficients are contained a table stored 
beforehand and in an input digital image signal for every class, To a noticed picture 
element, spatially and/or in time by primary combination of a class prediction 
coefficient from two or more nearby reference pixels and tables. By forming a 
predicted value of predetermined picture element data of an input digital image signal, 
and identifying a class prediction coefficient which produces a true value of a 
predetermined pixel, and the nearest predicted value. It is a digital image signal 
processor which consists of a means for generating a value for data prediction for 
creating a value of a noticed picture element for every class determined as a class 
sorting means for determining a class of a noticed picture element. 



[0011]In a digital image signal processor which needs for the invention according to 
claim 2 to create a value of a noticed picture element using two or more pixels of a 
noticed picture element which exist in the neighborhood spatially and/or in time. Are a 
learning device for determining two or more class prediction coefficients for every 
class, and it is contained in an input digital image signal, and spatially and/or in time to 
a noticed picture element by primary combination of two or more nearby reference 
pixels and a class prediction coefficient. When a predicted value of predetermined 
picture element data of an input digital image signal is formed, as an error of a 
predicted value and a true value of a predetermined pixel is made into the minimum, it 
is the learning device which determined a class prediction coefficient with the least 
square method. 

[0012]In a digital image signal processor for the invention according to claim 3 to 
change the 1 st picture signal with low resolution into the 2nd higher-resolution picture 
signal. As opposed to a noticed picture element in a table where two or more class 
prediction coefficients were beforehand stored for every class, and the 2nd picture 
signal, Spatially and/or in time by primary combination of a class prediction coefficient 
from a reference pixel and a table in two or more 1 st [ of the neighborhood ] picture 
signals. By forming a predicted value of predetermined picture element data of the 1st 
picture signal, and identifying a class prediction coefficient which produces a true 
value of a predetermined pixel, and the nearest predicted value. Class determination 
by a means and a class sorting means for generating a value for data prediction for 
creating a value of a noticed picture element for every class determined as a class 
sorting means for determining a class of a noticed picture element is answered, It is a 
digital image signal processor which consists of a means for creating a value of a 
noticed picture element using data ** data of a determined class. 
[0013] 

[Function]A noticed picture element can be spatial and/or a class division can be 
performed on a time target with reference to two or more nearby pixels according to 
the feature of a local picture. When a predicted value is formed by primary linearity 
combination with the input picture signal itself and the prediction coefficient for a 
class division, a class is determined corresponding to the prediction coefficient which 
an error with a true value makes produce the minimum predicted value. Even if an 
exact class division is meant and it increases the number of reference pixels, the 
capacity of the memory which memorizes the data for class sorting does not become 
so large. 
[0014] 

[Example]The example which applied this invention hereafter to the rise conversion 
which changes the definition of a video signal into a higher thing from a standard thing 
is explained. Application of the processing for interpolating the pixel thinned out in 
processing of subsampling etc. besides this application, the processing to which a 



picture is expanded electronically, the processing which detects the scene change in 
a video signal, the processing which forms the predicted value in DPCM, etc. Is 
possible for this invention. 

[0015]In drawing 1 , 1 is an input terminal of the digital image signal (an SD signal is 
called) of a standard definition (for example, the present NTSC system). Specifically, 
the regenerative signal from transmission by broadcast etc., VTR, etc. is supplied to 
the input terminal 1 . 2 is a scanning conversion circuit for changing an input signal into 
the signal of a block structure. 

[0016]The output signal dO of the scanning conversion circuit 2 is supplied to the 
class prediction device 3, the class determining circuit 4, and the output prediction 
device 6. In the class prediction device 3, provisional, or first stage the class 
prediction coefficient d2 and the signal dO from the table 5 with which the prediction 
coefficient is stored are calculated, and the predicted value d1 of an SD signal is 
generated. This predicted value d1 is supplied to the class determining circuit 4. 
[001 7]In the class determining circuit 4, a class is determined from comparison with 
the predicted value d1 and the true value dO of an SD signal. As an example, the class 
prediction coefficient which produces the predicted value d1 of the minimum 
[ prediction error ] absolutely with the true value dO. and a corresponding thing are 
determined as a class of the SD signal. The table 5 is referred to in the case of the 
determination of this class, and the prediction coefficient of the class of plurality (n) is 
supplied to the class determining circuit 4 from the table 5 one by one. The output 
prediction device 6 generates an HD signal by the operation of the data prediction 
coefficient d3 of a class and SD signal dO which were determined. This HD signal is 
taken out by the output terminal 7. The class prediction coefficient and data 
prediction coefficient which were beforehand gained by study are stored in the table 5. 
[0018]With reference to drawing 2 , an example of class determination and data 
prediction is explained. In drawing 2 . the value of the reference pixel which is 16 pieces 
other than the pixel (the value is sdg) of the center which is an object of prediction 
among 17 SD pixels (each value is expressed as sd, - sd^y) contained to SD prediction 
tap field is used, and predicted value sd' of SD pixel is formed. That is. if predicted 
value sd' expresses a class prediction coefficient as k, - k,, (however, except for kg), it 
will be generated by the following operation. 

sd-k^xsd,+ — +k7xsd7+k9xsd9+ ... +ki7xsdi7 (1)[0019]An above-mentioned estimate 
formula is related with one class, and the class prediction coefficient beforehand 
determined about n classes of zero to n-1 is stored in the table 5. Although the 
example of drawing 2 is the arrangement of a one-dimensional pixel, it may predict 
using two-dimensional pixel arrangement. Although the number of reference pixels of 
not being restricted to 16 is natural, even if this invention has many reference pixels, it 
has an advantage which the data volume of a class prediction coefficient does not 
increase too much. 



[0020]Data prediction of the HD picture element made in the output prediction device 
6 is made using three SD pixels near the position of the HD picture element to predict. 
In the example of drawing 2 , if a data prediction coefficient is made into w^, Wg. and W3, 
value hd' of an HD picture element will be generated by the following operation, 
hd -Wixsdy+Wgxsdg+Waxsdg (2) [0021]An above-mentioned estimate formula is related 
with one class, and the data prediction coefficient beforehand determined about n 
classes of zero to n-1 is stored in the table 5. That is, as the table 5 is shown in 
drawing 3 , the class prediction coefficient and data prediction coefficient of each 
class of zero to n-1 are stored. In the class determining circuit 4, first using the class 
prediction coefficient of the class 0. predicted value sd' is formed and the absolute 
value of the error of this and a true value is calculated by a formula (1). Hereafter, also 
about the other class, the absolute value of the error of a predicted value is calculated 
similarly, and this determines the minimum thing as a class of the SD pixel. 
[0022]The class prediction coefficient in the above-mentioned table 5 is beforehand 
determined by study. Drawing 4 is a flow chart which shows the processing at the time 
of study. The picture of a standard pattern is used as an example. Control of learning 
processing is started from Step 11, and the learned data corresponding to a known 
picture is formed in learned data formation of Step 12. As mentioned above, 
specifically, let arrangement of 1 7 pixels arranged as shown in drawing 2 be learned 
data of a lot. 

[0023]In the end of data of Step 13, if processing of all the inputted data, for example, 
data of one frame, is completed and it will not have ended to the prediction coefficient 
determination of Step 1 5, control shifts to normal equation generation of Step 1 4. In 
normal equation generation of Step 14, the normal equation of the equation (8) 
mentioned later, an equation (9), and an equation (10) is created. 

[0024]Control moves from the end of data of Step 13 to Step 15 after completing 
processing of all the data, in the prediction coefficient determination of Step 15, the 
formula (10) mentioned later is solved using a procession solution, and the prediction 
coefficients k1-k16 are determined. In the following step 16, by primary linearity 
combination (above-mentioned formula (1)) with coefficient k, - k^^, and value sd, of 
SD pixel - sdi7 that were determined. Predicted value sd' is calculated and the 
absolute value of the error between this prediction sd' and true-value sdg is calculated. 
An operation with error is performed about SD pixel used for determining a coefficient, 
and all the other SD pixels. There are very few errors about SD pixel used for 
determining a coefficient. 

[0025]The absolute value with error and threshold Th which were calculated are 
compared by the following step 17. If an absolute value with error is less than 
threshold Th, a class prediction coefficient is stored in a memory as a coefficient of 
the class i (Step 18). And it is determined at Step 19 whether to be i=n or not, and if 
that is right, learning processing is completed, and when that is not right, it will 



9|c9ic9K*9|e9|e*3|e9|cHeaN3|c)|e9|e i (Step 20). And it rctums to Step 12 and above-mentioned 
processing is repeated. 

[0026] However, in Step 17, the data of SD pixel in which an error becomes more than 
threshold Th is distinguished, and what is used as learned data is limited to what 
produced the error more than threshold Th in Step 21 of data selection. Thus, each 
class prediction coefficient of n-1 is determined from the class 0. 
[0027]Processing of Step 14 (normal equation generation) in drawing 4 and Step 15 
(prediction coefficient determination) is explained more to details. True-value sd8 of 
an attention SD pixel is expressed as y, the point estimate sd' is expressed as y', and 
it is n (in drawing 3 .) of the circumference. Primary linearity joint y-kiXi+k2X2+ of n tap 
according to coefficient - k„ the whole class when the value of n= 16 pixel is made 
into X, - x„ .... +k„x„ (3) 

It sets up. kj is an undetermined coefficient before study. 

[0028]As mentioned above, it is made for every class, when a data number is m, a 
formula (3) is followed, and study is y/=kiXji+k2Xj2+.... +k„Xj„ (4) 
(However, j= 1 , 2 m) 

[0029]Since it is not decided in m>n that k, - k„ will be meaning, it is an element of the 
error vector E ej=yj - (kiXji+k2Xj2+ .... +k„Xj„) (5) 

It is defined as (however, j= 1, 2 m), and the coefficient which makes the following 

formula (6) the minimum is calculated. 

[0030] 

[Equation 1] 



[0031 ]It is a solution what is called by a least square method. The partial differential 
coefficient by kj of the formula 4 is calculated here. 
[0032] 
[Equation 2] 
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[0033]Because what is necessary is just to decide k j each to set a formula (1 1) to 0, 

[0034] 

[Equation 3] 
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[0035]It is if it carries out and a procession is used, [0036] 
[Equation 4] 
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[0037]It becomes. Generally this equation is called the normal equation. If this 
equation is swept out and it solves about kj using general procession solutions, such 
as law, prediction coefficient can be found. 

[0038]The method for determining an above-mentioned prediction class is an example, 
and various modification is possible for it. 

[0039]The data prediction coefficient wi is beforehand determined by study using the 
HD signal and the SD signal acquired from this. Drawing 5 is a flow chart for data 
prediction. Control of learning processing is started from Step 31, and the learned 
data corresponding to a known picture is formed in learned data formation of Step 32. 
As mentioned above, specifically, three SD pixels and one HD picture element are 
learned data of a lot like the arrangement of drawing 2 . In the end of data of Step 33, if 
processing of all the inputted data, for example, data of one frame, is completed and it 
will not have ended to the prediction coefficient determination of Step 36, control 
shifts to the class determination of Step 34. 

[0040]The class determination of Step 34 is the same processing as the 
above-mentioned class determining circuit 4. That is, a predicted value is formed by 
primary linearity combination with the class prediction coefficient determined as 
mentioned above by study, and the picture element data of an SD signal, and the error 
of this and a true value discriminates from the class to which the class prediction 
coefficient which produces the minimum predicted value belongs. A normal equation is 
created in normal equation generation of the following step 35. 

[0041]Control moves from the end of data of Step 33 to Step 36 after completing 



processing of all the data, in the prediction coefficient determination of Step 36, it 
solves using a procession solution and the data prediction coefficient w is decided. A 
data prediction coefficient is stored in a memory at the prediction coefficient store of 
Step 37. and control of learning processing is completed at Step 38. Normal equation 
generation of Step 35 and the prediction coefficient determination of Step 36 are 
based on the least square method like an above-mentioned class prediction 
coefficient. 

[0042]The central value beforehand formed by study can be used without using a data 
prediction coefficient, in order to generate an HD picture element. Drawing 6 shows an 
example of the hardware at the time of study. The digital HD signal for study is 
supplied from the input terminal shown by 41. The still picture signal of a different 
pattern can be used as this input signal. An SD signal is formed when an HD signal 
passes the level infanticide circuit 42 and the vertical infanticide circuit 43. 
[0043]The generated SD signal is supplied to the class sorting circuit 44. The class 
sorting circuit 44 consists of a scanning conversion circuit, a class prediction device, 
a table where the class prediction coefficient was stored, and a class determining 
circuit like the composition of drawing 1 . The class code generated in the output of 
the class sorting circuit 44 is supplied as an address to the frequency memory 45 and 
the data memory 46. These memories 45 and 46 are cleared before the study start. 
[0044]An input HD signal is supplied to the divider 49 as a dividend via the delay 
circuit 47 and the adder circuit 48. and let the output signal (quotient of division) of 
the divider 49 be input data of the data memory 46. The delay circuit 47 is for delaying 
time required for class sorting, and data. 

[0045]The read-out output of the frequency memory 45 is supplied to the multiplier 
50 and the +1 circuit 51. +The output of the one circuit 51 is made into the data input 
of the frequency memory 45, and the divider 49 is supplied as a divisor. If an address is 
specified by a class code, the contents of the address will be read and, as for the 
frequency memory 45 and the data memory 46. frequency and data will be written in 
to the address, respectively. +Cumulative frequency is memorized in each address of 
the frequency memory 45 by the one circuit 51. 

[0046]In the composition of drawing 6 , if a certain class code occurs, the cumulative 
frequency and the central value of the class will be read from the frequency memory 
45 and the data memory 46. respectively, and multiplication will be carried out with the 
multiplier 50. Therefore, an accumulation central value occurs from the multiplier 50. 
This accumulation central value and the present central value from the delay circuit 
47 are added in the adder circuit 48. An added result is supplied to the divider 49. the 
central value in consideration of the present central value is formed, and this is 
written in the data memory 46. 

[0047]The accuracy of a central value can be raised by repeating about the learned 
data into which this processing is inputted. The central value of each class stored in 



the data memory 46 is used for rise conversion. Drawing 7 shows the contents of the 
table when using a central value. The central value LO determined with the 
composition of the class coefficient and drawing 6 w hich were mentioned above - 
Ln-1 are stored in each of the class zero to n-1. 

[0048]The processing which determines a central value is realizable not only by the 
hardware constitutions which show the example of drawing 6 but software processing, 
[0049] 

[Effect of the Invention]In this invention, according to class sorting, the local features 
of an SD signal are extracted and the HD signal specified according to it is outputted. 
Therefore, rise conversion which increases resolution can be performed good. 
In this invention, when a predicted value is generated and the error of this predicted 
value and true value detects the minimum thing by two or more reference pixels and 
primary linearity combination of a prediction coefficient, class sorting is performed. 
Therefore, since a prediction coefficient equal to the number of reference pixels is 
memorized, even if it increases the number of reference pixels, there is an advantage 
to which the capacity of the memory which stores the table for class sorting does not 
increase so much. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram of one example which applied this invention to the 
device for performing rise conversion. 

[Drawing 2] It is an approximate line figure showing the class sorting in this invention, 
and the arrangement of the pixel for data prediction. 

[Drawing 3] It is an approximate line figure showing the composition of the table where 
a class prediction coefficient and a data prediction coefficient are stored. 
[Drawing 4] It is a flow chart when performing study for determining a class prediction 
coefficient by software processing. 

[Drawing 5] It is a flow chart when performing study for determining a data prediction 
coefficient by software processing. 

[Drawing 6] It is a block diagram of an example of the composition at the time of the 
study for [ which determines the central value for data prediction ] carrying out. 
[Drawing 7] It is an approximate line figure showing the composition of the table where 
a class prediction coefficient and a central value are stored. 
[Description of Notations] 

3 Class prediction device 

4 Class determining circuit 



1 

\ 

5 The table where the class prediction coefficient and the data prediction coefficient 
were stored 



